{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "<a href=\"https://colab.research.google.com/drive/1aaU4YZC-fswSImo1fV-w67FXPQg5Ictm?usp=sharing\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>"
      ],
      "metadata": {
        "id": "AWgDx0D0ED7z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 📊 What is Vector Embeddings?\n",
        "\n",
        "Vector embedding is a way to represent words, phrases, or texts as numerical vectors in a multi-dimensional space. This helps the model understand language better by capturing meanings and relationships between words.\n",
        "\n",
        "![Vector Embedding](https://qdrant.tech/articles_data/what-are-embeddings/BERT-model.jpg)\n",
        "Source: [Qdrant Blog](https://qdrant.tech/articles/what-are-embeddings/)\n",
        "\n",
        "\n",
        "\n",
        "#### 📌 Key Points\n",
        "\n",
        "1. **Representation:** Each word or token is a vector of real numbers.\n",
        "2. **Dimensionality:** These vectors usually have hundreds or thousands of dimensions, where a dimension represents a specific feature or characteristic of the word (e.g., meaning, context, or usage).\n",
        "3. **Semantic Meaning:** Similar words are closer together in this space.\n",
        "\n",
        "\n",
        "#### 🔄 Example: Animal Relationship Analogy 🐶❤️🐱\n",
        "\n",
        "1. Consider a simplified 3D vector space with these word embeddings:\n",
        "\n",
        "  * cat: [2, 4, -1]\n",
        "  * kitten: [1.5, 3, -0.5]\n",
        "  * dog: [-2, 3, 1]\n",
        "  * puppy: [-1.5, 2, 0.5]\n",
        "\n",
        "2. **Explanation**\n",
        "\n",
        "  * **Semantic Similarity:**\n",
        "    * \"**Cat**\" and \"**kitten**,\" as well as \"**dog**\" and \"**puppy**,\" have approximate similar type vectors, reflecting their semantic (similarity) closeness.\n",
        "\n",
        "  * **Analogy Relationship:** For \"cat is to kitten as dog is to what?\"\n",
        "\n",
        "    * **We can solve this by vector arithmetic**\n",
        "\n",
        "      [dog] - [cat] + [kitten] ≈ [puppy]\n",
        "    \n",
        "    * **Let's calculate**\n",
        "\n",
        "      (−2, 3, 1) **−** (2, 4, −1) **+** (1.5, 3, −0.5) **≈** −1.5, 2, 0.5 (close to \"puppy\").\n",
        "\n",
        "    * 🤔💡 **Understanding above Vector Arithmetic**\n",
        "\n",
        "      In the analogy \"cat is to kitten as dog is to what?\" we want to find a word that fits the same relationship.\n",
        "\n",
        "      Here's a simpler way to think about the vector arithmetic:\n",
        "\n",
        "      * **Think of Relationships**:\n",
        "\n",
        "        * \"Cat\" is a more general term, while \"kitten\" is a specific younger version of \"cat.\"\n",
        "\n",
        "        * Similarly, \"dog\" is the general term, and we want to find the younger version of \"dog.\"\n",
        "\n",
        "      * **Why the Math?**\n",
        "\n",
        "        When we say [dog]−[cat]+[kitten], we are using the relationship between the words:\n",
        "          * Subtracting the vector for \"cat\" from \"dog\" helps us find out what makes \"dog\" different from \"cat.\" This captures the essence of being a dog instead of a cat.\n",
        "        \n",
        "          * Adding the vector for \"kitten\" represents moving back towards the younger version of a dog, just as \"kitten\" is the younger version of \"cat.\"\n",
        "\n",
        "      * **Result:**\n",
        "\n",
        "        The result of this calculation gives us a vector that points to a word that shares a similar relationship with \"dog\" as \"kitten\" does with \"cat.\" In this case, that word is \"puppy.\"\n",
        "\n",
        "      So, in simple terms, we're using vector math to navigate the relationships between these words and find the one that fits the analogy.\n",
        "\n",
        "  * [**Cosine Similarity:**](https://www.youtube.com/watch?v=e9U0QAFbfLI) Measures similarity between words, with higher values indicating similar meanings.\n",
        "\n",
        "  * **Capturing Relationships:** Embeddings encode relationships like animal type, age, and size etc.\n",
        "\n",
        "  * **Contextual Usage:** In advanced embedding models, word embeddings may vary by context, capturing similarity meanings.\n",
        "\n",
        "This shows how vector embeddings can encode complex semantic relationships, enabling mathematical operations to reveal meaningful linguistic patterns, foundational for understanding and generating text in Large Language Models.\n",
        "\n",
        "[Qdrant Blog](https://qdrant.tech/articles/what-are-embeddings/)\n",
        "\n",
        "\n",
        "--------------------------------------------------------------------------\n",
        "\n",
        "\n",
        "### 📚 What is Embedding Models?\n",
        "\n",
        "Embedding models are machine learning algorithms that convert text, images, or other data into dense vector representations in a high-dimensional space. These representations capture semantic meaning and relationships between items.\n",
        "\n",
        "#### 📌 Key points\n",
        "\n",
        "1. Transform data into numerical vectors\n",
        "2. Preserve semantic similarity in vector space\n",
        "3. Useful for various downstream tasks\n",
        "\n",
        "#### 🔄 Examples\n",
        "\n",
        "1. **Text embeddings**\n",
        "\n",
        "  * OpenAI's Embedding Model\n",
        "  * Google's Embedding Model\n",
        "  * Huggingface open-source Embedding Models\n",
        "\n",
        "2. **Image embeddings**\n",
        "\n",
        "  * OpenAI's CLIP (Contrastive Language-Image Pre-Training)\n",
        "\n",
        "3. [**Multi-modal embeddings**](https://weaviate.io/blog/multimodal-models)\n",
        "\n",
        "  * Amazon Titan Multimodel\n",
        "  * Google's Multimodel Embedding API\n",
        "  * Microsoft's Azure Multimodel Embedding API\n",
        "  * multi2vec-clip\n",
        "\n",
        "These models are used in search, recommendation systems, clustering, and as input for other machine learning tasks.\n",
        "\n",
        "\n",
        "--------------------------------------------------------------------------\n",
        "\n",
        "\n",
        "### 🏆 Where to find \"RIGHT\" Embedding Models based on Ranking?\n",
        "\n",
        "1. The Masstive Text Embedding Benchmark (MTEB) Leaderboard on Hugging Face is an excellent resource for finding the latest proprietary and open-source text embedding models, along with their performance statistics for tasks like **retrieval** and **summarization**.\n",
        "\n",
        "  It’s important to see benchmark results with care, as they are self-reported and may use publicly available datasets, so it's advisable to evaluate and experiment with popular embedding models on your own data and see the best fit.\n",
        "\n",
        "  General strategy is to start with most popular Embedding Models. You can find the top-10 models to use based on the anonymous ranking at [MTEB Arena Leaderboard](https://huggingface.co/spaces/mteb/arena)\n",
        "\n",
        "2. Key metrics to focus on include the **\"average\"** and **\"retrieval average\"** scores, as well as the model's **Size** (manageable on consumer hardware), **Max Tokens** (ideally around 100-200 tokens for embedding), and **Embedding Dimensions** (ideally up to 512 tokens balancing detail capture and operational efficiency).\n",
        "\n",
        "3. The leaderboard features a mix of **Small**, **Large**, **Proprietary**, and **Open-Source** models for comparison.\n",
        "\n",
        "[MTEB Leaderboard Link](https://huggingface.co/spaces/mteb/leaderboard)\n",
        "\n",
        "[MTEB Arena Leaderboard: Retrieval](https://huggingface.co/spaces/mteb/arena)\n"
      ],
      "metadata": {
        "id": "HxxOWz9dpsYj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Install required libraries"
      ],
      "metadata": {
        "id": "Y3axTI0sp5Hg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -qU \\\n",
        "     Sentence-transformers==3.4.1 \\\n",
        "     langchain==0.3.19 \\\n",
        "     langchain-openai==0.3.7 \\\n",
        "     langchain-google-genai==2.0.11 \\\n",
        "     langchain-huggingface==0.1.2 \\\n",
        "     langchain-chroma==0.2.2 \\\n",
        "     langchain-community==0.3.18 \\\n",
        "     einops==0.8.1"
      ],
      "metadata": {
        "id": "ShxTNxM5gqtr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "46dcb38d-e4e9-4d5b-9bef-9dbb1b32753b",
        "collapsed": true
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/67.3 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.3/67.3 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.3/55.3 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.5/2.5 MB\u001b[0m \u001b[31m58.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m611.1/611.1 kB\u001b[0m \u001b[31m44.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.4/2.4 MB\u001b[0m \u001b[31m88.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m70.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m63.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m363.4/363.4 MB\u001b[0m \u001b[31m4.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.8/13.8 MB\u001b[0m \u001b[31m82.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m24.6/24.6 MB\u001b[0m \u001b[31m16.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m883.7/883.7 kB\u001b[0m \u001b[31m45.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m664.8/664.8 MB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.5/211.5 MB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.3/56.3 MB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m127.9/127.9 MB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m207.5/207.5 MB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.1/21.1 MB\u001b[0m \u001b[31m62.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m284.2/284.2 kB\u001b[0m \u001b[31m21.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m94.9/94.9 kB\u001b[0m \u001b[31m8.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m62.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.9/50.9 kB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m101.6/101.6 kB\u001b[0m \u001b[31m9.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.3/13.3 MB\u001b[0m \u001b[31m85.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.9/55.9 kB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m177.4/177.4 kB\u001b[0m \u001b[31m17.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m65.0/65.0 kB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m118.7/118.7 kB\u001b[0m \u001b[31m12.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.6/76.6 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m319.7/319.7 kB\u001b[0m \u001b[31m29.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.3/62.3 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m459.8/459.8 kB\u001b[0m \u001b[31m40.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.0/72.0 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.0/4.0 MB\u001b[0m \u001b[31m89.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m452.6/452.6 kB\u001b[0m \u001b[31m34.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for pypika (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "google-generativeai 0.8.4 requires google-ai-generativelanguage==0.6.15, but you have google-ai-generativelanguage 0.6.16 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Import related libraries"
      ],
      "metadata": {
        "id": "9jJ1vqs-p_Zx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import getpass\n",
        "\n",
        "from langchain_google_genai import (\n",
        "    ChatGoogleGenerativeAI,\n",
        "    HarmBlockThreshold,\n",
        "    HarmCategory,\n",
        ")\n",
        "from langchain.vectorstores import Chroma\n",
        "from langchain_community.document_loaders import WebBaseLoader\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "from langchain.prompts import PromptTemplate\n",
        "from langchain.chains import RetrievalQA\n",
        "from sklearn.metrics.pairwise import cosine_similarity"
      ],
      "metadata": {
        "id": "RL-3LsYogoH5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Provide OpenAI API Key.\n",
        "\n",
        "If you want to use OpenAI Embedding. You can create OpenAI API key using following link\n",
        "\n",
        "- [OpenAI API Key](https://platform.openai.com/settings/profile?tab=api-keys)"
      ],
      "metadata": {
        "id": "Qa2FMZe4341-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass()"
      ],
      "metadata": {
        "id": "kdOFTZoS35DJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "54896bd6-cd3e-4b96-ec2c-d01b59a6a979"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Provide Google API Key.\n",
        "\n",
        "It can be used both for Gemini Pro LLM  & Google Embedding Model. You can create Google API key using following link\n",
        "\n",
        "- [Google Gemini-Pro API Key](https://console.cloud.google.com/apis/credentials)\n",
        "\n",
        "- [YouTube Video explaining Google API Key](https://www.youtube.com/watch?v=ZHX7zxvDfoc)\n",
        "\n"
      ],
      "metadata": {
        "id": "F6UeDlrgqI2A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "os.environ[\"GOOGLE_API_KEY\"] = getpass.getpass()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yobvrD3glfd4",
        "outputId": "4d42d340-708a-4f68-c0bc-48f20dd3f3bf"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Provide Huggingface API Key.\n",
        "\n",
        "If you want to use Huggingface Embedding Models. You can create Huggingface API key using following link\n",
        "\n",
        "- [Huggingface API Key](https://huggingface.co/settings/tokens)\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "S1dLpYboqeIK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "os.environ[\"HF_TOKEN\"] = getpass.getpass()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TQ6scBGZlhpG",
        "outputId": "3970916a-dd19-45ad-b8d1-9cc98601f030"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Embedding Model Decision Flow\n",
        "\n",
        "<br>\n",
        "\n",
        "![MTEB Areana](https://raw.githubusercontent.com/genieincodebottle/generative-ai/main/images/embedding.png)"
      ],
      "metadata": {
        "id": "_rzpVybYSwZ4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 💰 Paid: OpenAI Embedding Model\n",
        "\n",
        "1. 🤖 **Models:** text-embedding-3-small, text-embedding-3-large, ada v2\n",
        "2. 👍 **Pros:**\n",
        "  * High quality embeddings\n",
        "  * Multiple model options for different needs\n",
        "  * Easy integration with OpenAI API\n",
        "  * Batch processing available for cost savings\n",
        "  * Regular updates and improvements\n",
        "  * Suitable for various NLP tasks\n",
        "3. 👎 **Cons:**\n",
        "  * Not free - usage costs can add up\n",
        "  * Closed-source model\n",
        "  * Requires API key and internet connection\n",
        "  * Limited control over model architecture\n",
        "  * Potential privacy concerns with data handling\n",
        "  * Usage subject to OpenAI's terms and policies\n",
        "4. 🔗 [Document Link](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings)\n"
      ],
      "metadata": {
        "id": "6mAlfozR4YBg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_openai import OpenAIEmbeddings\n",
        "\n",
        "def get_openai_embeddings():\n",
        "    openai_embeddings = OpenAIEmbeddings(model=\"text-embedding-3-large\")\n",
        "    return openai_embeddings"
      ],
      "metadata": {
        "id": "EzdspqyR4YK4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 🔓 Free: Google Gemini's Embedding Model\n",
        "\n",
        "1. 🤖**Models:** text-embedding-004\n",
        "2. 👍 **Pros:**\n",
        "  * Free to use\n",
        "  * Supports 100+ languages\n",
        "  * Long context window (3,072 tokens)\n",
        "  * Multiple embedding dimensions (768 and 1,024)\n",
        "  * High-quality embeddings\n",
        "  * Easy integration with Google Cloud\n",
        "3. 👎 **Cons:**\n",
        "  * Potential usage limits or quotas\n",
        "  * Closed-source model\n",
        "  * Possible vendor lock-in to Google ecosystem\n",
        "  * Limited customization options\n",
        "  * Dependency on Google's infrastructure\n",
        "  * Potential for unexpected changes to service terms\n",
        "4. 🔗 [Document Link](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings-api#generative-ai-get-text-embedding-python_vertex_ai_sdk)\n"
      ],
      "metadata": {
        "id": "wY2uCq0XBeXH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_google_genai import GoogleGenerativeAIEmbeddings\n",
        "\n",
        "def get_google_embeddings():\n",
        "    gemini_embeddings = GoogleGenerativeAIEmbeddings(model=\"models/text-embedding-004\")\n",
        "    return gemini_embeddings"
      ],
      "metadata": {
        "id": "3E4nq2tvBepo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 🔓 Free: Huggingface Open-Source Embedding Models\n",
        "\n",
        "1. 🤖 **Models:** gte-large-en-v1.5, bge-multilingual-gemma2,snowflake-arctic-embed-l, nomic-embed-text-v1.5, e5-mistral-7b-instruct and more. See the [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) to check top open-source embedding models as per your usecase.\n",
        "2. 👍 **Pros:**\n",
        "  * Open-source and freely available\n",
        "  * Flexible and customizable\n",
        "  * Community support and updates\n",
        "  * Integration with Hugging Face ecosystem\n",
        "  * Suitable for various NLP tasks\n",
        "  * Can be fine-tuned on specific domains\n",
        "3. 👎 **Cons:**\n",
        "  * Potentially less optimized than commercial alternatives\n",
        "  * Performance can vary depending on specific use case\n",
        "  * Limited official support compared to commercial options\n",
        "  * May need significant computational resources for training/fine-tuning\n",
        "  * Ongoing maintenance and updates depend on community involvement\n",
        "4. 🔗 **MTEB Leaderboard:**\n",
        "\n",
        "  * [Huggingface Embedding Models Leaderboard](https://huggingface.co/spaces/mteb/leaderboard)\n",
        "\n",
        "  * [Massive Text Embedding Benchmark (MTEB)Arena Leaderboard](https://huggingface.co/spaces/mteb/arena)"
      ],
      "metadata": {
        "id": "l3sbWOlACRKk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Huggingface based Nomic AI Embedding model\n",
        "\n",
        "You can use any other huggingface open-source embedding models as per your requirement, fitness and system constraints. You can get the model name from MTEB leaderboard.\n",
        "\n",
        "Popular Models\n",
        "\n",
        "1. nomic-ai/nomic-embed-text-v1.5\n",
        "2. nomic-ai/nomic-embed-text-v1\n",
        "3. sentence-transformers/all-MiniLM-L12-v2\n",
        "4. sentence-transformers/all-MiniLM-L6-v2\n",
        "\n",
        ".....\n",
        "\n"
      ],
      "metadata": {
        "id": "R1JkUYs5pCsq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_huggingface import HuggingFaceEmbeddings\n",
        "\n",
        "def get_huggingface_embeddings():\n",
        "\n",
        "    # Change model_name as per your choosen huggingface embedding model\n",
        "    nomic_embeddings = HuggingFaceEmbeddings(model_name=\"nomic-ai/nomic-embed-text-v1.5\", model_kwargs = {'trust_remote_code': True})\n",
        "    return nomic_embeddings"
      ],
      "metadata": {
        "id": "ERz6lnISCRSy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Let's implement Basic RAG using following components\n",
        "\n",
        "1. **Chroma:** Used as the vector store for efficient similarity search.\n",
        "2. **Embedding Models:** As we choose OpenAI, Google or Huggingface Embedding Models\n",
        "3. **ChatGoogleGenerativeAI:** The Gemini Pro model used for generating responses.\n",
        "4. **cosine_similarity:** Used for computing similarities in the evaluation step."
      ],
      "metadata": {
        "id": "SfvFZd4y-962"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Helper function for printing docs\n",
        "def pretty_print_docs(docs):\n",
        "    # Iterate through each document and format the output\n",
        "    for i, d in enumerate(docs):\n",
        "        print(f\"{'-' * 50}\\nDocument {i + 1}:\")\n",
        "        print(f\"Content:\\n{d.page_content}\\n\")\n",
        "        print(\"Metadata:\")\n",
        "        for key, value in d.metadata.items():\n",
        "            print(f\"  {key}: {value}\")\n",
        "    print(f\"{'-' * 50}\")  # Final separator for clarity\n",
        "\n",
        "# Example usage\n",
        "# Assuming `docs` is a list of Document objects"
      ],
      "metadata": {
        "id": "gKsdj7SBCNWS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 1: Load and preprocess data code"
      ],
      "metadata": {
        "id": "ksRC4Oj6BRxX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def load_and_process_data(url):\n",
        "    # Load data from web\n",
        "    loader = WebBaseLoader(url)\n",
        "    data = loader.load()\n",
        "\n",
        "    # Split text into chunks (Experiment with Chunk Size and Chunk Overlap to get optimal chunking)\n",
        "    text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)\n",
        "    chunks = text_splitter.split_documents(data)\n",
        "\n",
        "    # Add unique IDs to each text chunk\n",
        "    for idx, chunk in enumerate(chunks):\n",
        "        chunk.metadata[\"id\"] = idx\n",
        "\n",
        "    return chunks"
      ],
      "metadata": {
        "id": "0Uk2WjeyBR9Q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 2: Intialize Embedding Model"
      ],
      "metadata": {
        "id": "F6-IdSjk8IhJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_embedding_model(embedding_model=\"huggingface\"):\n",
        "\n",
        "    if embedding_model == \"openai\":\n",
        "        embeddings = get_openai_embeddings()\n",
        "    elif embedding_model == \"google\":\n",
        "        embeddings = get_google_embeddings()\n",
        "    elif embedding_model == \"huggingface\":\n",
        "        embeddings = get_huggingface_embeddings()\n",
        "    else:\n",
        "        embeddings = get_huggingface_embeddings()\n",
        "    return embeddings"
      ],
      "metadata": {
        "id": "CJ5LYqIW8Iox"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 3: Create Chroma vector store"
      ],
      "metadata": {
        "id": "K-y-ZdbZBVM1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def create_vector_stores(chunks, embeddings):\n",
        "    # Create vector stores using the specified embedding model\n",
        "    vectorstore = Chroma.from_documents(chunks, embeddings)\n",
        "    return vectorstore"
      ],
      "metadata": {
        "id": "o5VTvGLxBVVr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 4: Implement Basic RAG\n",
        "\n",
        "1. **Retrieval:** We use the Chroma vector store's similarity search to retrieve the top 5 most relevant documents for the query.\n",
        "2. **Context Formation:** We combine the retrieved documents into a single context string.\n",
        "3. **Response Generation:** Using the Gemini Pro model, we generate a final response based on the retrieved context and the query."
      ],
      "metadata": {
        "id": "BcrHq-WF14Qi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def basic_rag(query, vectorstore, llm):\n",
        "    # Retrieve relevant documents\n",
        "    docs = vectorstore.similarity_search(query, k=5)\n",
        "    context = \"\\n\\n\".join([doc.page_content for doc in docs])\n",
        "\n",
        "    # Generate response\n",
        "    prompt = f\"{context}\\n\\nQuestion: {query}\\nAnswer:\"\n",
        "    response = llm.invoke(prompt)\n",
        "\n",
        "    return {\n",
        "        \"query\": query,\n",
        "        \"final_answer\": response.content,\n",
        "        \"retrieval_method\": \"Basic similarity search\",\n",
        "        \"context\": context\n",
        "    }"
      ],
      "metadata": {
        "id": "VDrqbav-131u"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 5: RAG Evaluation\n",
        "\n",
        "1. We compute embeddings for the query, response, and context.\n",
        "2. We calculate cosine similarities between query-response and response-context.\n",
        "3. We compute an overall relevance score as the average of these similarities.\n"
      ],
      "metadata": {
        "id": "oi2pnLV272Bx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def evaluate_response(query, embeddings, response, context):\n",
        "    # Compute embeddings\n",
        "    query_embedding = embeddings.embed_query(query)\n",
        "    response_embedding = embeddings.embed_query(response)\n",
        "    context_embedding = embeddings.embed_query(context)\n",
        "\n",
        "    # Compute cosine similarities\n",
        "    query_response_similarity = cosine_similarity([query_embedding], [response_embedding])[0][0]\n",
        "    response_context_similarity = cosine_similarity([response_embedding], [context_embedding])[0][0]\n",
        "\n",
        "    # Compute relevance score (average of the two similarities)\n",
        "    relevance_score = (query_response_similarity + response_context_similarity) / 2\n",
        "\n",
        "    return {\n",
        "        \"query_response_similarity\": query_response_similarity,\n",
        "        \"response_context_similarity\": response_context_similarity,\n",
        "        \"relevance_score\": relevance_score\n",
        "    }"
      ],
      "metadata": {
        "id": "YLZtruPH72MP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Step 4: Load and process data and create chunks to store in the Chroma Vector Store\n",
        "\n",
        "1. [Langchain Chunking Strategy](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/)\n",
        "2. [Langchain Vectorstore](https://python.langchain.com/v0.1/docs/modules/data_connection/vectorstores/)\n"
      ],
      "metadata": {
        "id": "D5cQdItbNMsl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Initialize the gemini-pro language model with specified settings (Change temeprature  and other parameters as per your requirement)\n",
        "llm = ChatGoogleGenerativeAI(model=\"gemini-pro\", temperature=0.3, safety_settings={\n",
        "          HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,\n",
        "        },)\n",
        "\n",
        "# Load Documents from a web URL\n",
        "url = \"https://en.wikipedia.org/wiki/Artificial_intelligence\"\n",
        "chunks = load_and_process_data(url)\n",
        "\n",
        "# Get Embedding Model.\n",
        "# embedding_model value options (openai, google, huggingface)\n",
        "embeddings = get_embedding_model(embedding_model=\"google\")\n",
        "\n",
        "# Create vector stores using the specified embedding model\n",
        "vectorstore = create_vector_stores(chunks, embeddings)"
      ],
      "metadata": {
        "id": "B5I5PCsKNP2N"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 5: RAG in Action to test Embedding Model\n",
        "\n",
        "## This implementation demonstrashowstes the key parts of Basic RAG with evaluation:\n",
        "\n",
        "1. Chunking and embedding of source documents\n",
        "2. Retrieval of relevant documents based on query similarity\n",
        "3. Generation of a response using the retrieved context\n",
        "4. Evaluation of the response based on its similarity to both the query and the context"
      ],
      "metadata": {
        "id": "PpXIJFEAlVsT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"What are the main applications of artificial intelligence in healthcare?\"\n",
        "result = basic_rag(query, vectorstore, llm)\n",
        "\n",
        "\n",
        "print(f\"Query: {result['query']}\")\n",
        "print(\"=========================\")\n",
        "print(f\"Final Answer: {result['final_answer']}\")\n",
        "print(\"=========================\")\n",
        "print(f\"Retrieval Method: {result['retrieval_method']}\")\n",
        "\n",
        "\n",
        "# Evaluate the response\n",
        "evaluation = evaluate_response(query, embeddings, result[\"final_answer\"], result[\"context\"])\n",
        "print(\"\\nEvaluation:\")\n",
        "print(f\"Query-Response Similarity: {evaluation['query_response_similarity']:.4f}\")\n",
        "print(f\"Response-Context Similarity: {evaluation['response_context_similarity']:.4f}\")\n",
        "print(f\"Overall Relevance Score: {evaluation['relevance_score']:.4f}\")"
      ],
      "metadata": {
        "id": "aIXlHaWqgcza",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9653662d-d681-402e-8377-f6738af58765"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Query: What are the main applications of artificial intelligence in healthcare?\n",
            "=========================\n",
            "Final Answer: The main applications of artificial intelligence in healthcare include:\n",
            "- Increasing patient care and quality of life\n",
            "- Processing and integrating big data for medical research\n",
            "- Overcoming discrepancies in funding allocated to different fields of research\n",
            "- Deepening the understanding of biomedically relevant pathways\n",
            "=========================\n",
            "Retrieval Method: Basic similarity search\n",
            "\n",
            "Evaluation:\n",
            "Query-Response Similarity: 0.8314\n",
            "Response-Context Similarity: 0.7585\n",
            "Overall Relevance Score: 0.7949\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Retrieval Demonstration:\n",
        "\n",
        "We showcase the retrieval process by displaying the retrieved documents for a sample query."
      ],
      "metadata": {
        "id": "dVOq9sU--Vj6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Demonstrate retrieval\n",
        "demo_query = \"Explain the concept of machine learning and its relationship to AI\"\n",
        "print(f\"\\nDemonstration Query: {demo_query}\")\n",
        "\n",
        "# Retrieve documents\n",
        "docs = vectorstore.similarity_search(demo_query, k=5)\n",
        "print(\"\\nRetrieved Documents:\")\n",
        "pretty_print_docs(docs)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UHu_H7gX0qRE",
        "outputId": "23c216d3-0de0-4ebc-9927-1161e271b395"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Demonstration Query: Explain the concept of machine learning and its relationship to AI\n",
            "\n",
            "Retrieved Documents:\n",
            "--------------------------------------------------\n",
            "Document 1:\n",
            "Content:\n",
            "There are several kinds of machine learning. Unsupervised learning analyzes a stream of data and finds patterns and makes predictions without any other guidance.[49] Supervised learning requires a human to label the input data first, and comes in two main varieties: classification (where the program must learn to predict what category the input belongs in) and regression (where the program must deduce a numeric function based on numeric input).[50]\n",
            "\n",
            "Metadata:\n",
            "  id: 40\n",
            "  language: en\n",
            "  source: https://en.wikipedia.org/wiki/Artificial_intelligence\n",
            "  title: Artificial intelligence - Wikipedia\n",
            "--------------------------------------------------\n",
            "Document 2:\n",
            "Content:\n",
            "Learning\n",
            "Machine learning is the study of programs that can improve their performance on a given task automatically.[46] It has been a part of AI from the beginning.[e]\n",
            "\n",
            "Metadata:\n",
            "  id: 39\n",
            "  language: en\n",
            "  source: https://en.wikipedia.org/wiki/Artificial_intelligence\n",
            "  title: Artificial intelligence - Wikipedia\n",
            "--------------------------------------------------\n",
            "Document 3:\n",
            "Content:\n",
            "^ DiFeliciantonio (2023).\n",
            "\n",
            "^ Goswami (2023).\n",
            "\n",
            "^ a b Turing (1950), p. 1.\n",
            "\n",
            "^ Turing (1950), Under \"The Argument from Consciousness\".\n",
            "\n",
            "^ Russell & Norvig (2021), p. 3.\n",
            "\n",
            "^ Maker (2006). sfnp error: no target: CITEREFMaker2006 (help)\n",
            "\n",
            "^ McCarthy (1999).\n",
            "\n",
            "^ Minsky (1986).\n",
            "\n",
            "^ \"What Is Artificial Intelligence (AI)?\". Google Cloud Platform. Archived from the original on 31 July 2023. Retrieved 16 October 2023.\n",
            "\n",
            "Metadata:\n",
            "  id: 371\n",
            "  language: en\n",
            "  source: https://en.wikipedia.org/wiki/Artificial_intelligence\n",
            "  title: Artificial intelligence - Wikipedia\n",
            "--------------------------------------------------\n",
            "Document 4:\n",
            "Content:\n",
            "^ \n",
            "Transfer learning:\n",
            "Russell & Norvig (2021, pp. 281)\n",
            "The Economist (2016)\n",
            "\n",
            "^ \"Artificial Intelligence (AI): What Is AI and How Does It Work? | Built In\". builtin.com. Retrieved 30 October 2023.\n",
            "\n",
            "^ \n",
            "Computational learning theory:\n",
            "Russell & Norvig (2021, pp. 672–674)\n",
            "Jordan & Mitchell (2015)\n",
            "\n",
            "^ \n",
            "Natural language processing (NLP):\n",
            "Russell & Norvig (2021, chpt. 23–24)\n",
            "Poole, Mackworth & Goebel (1998, pp. 91–104)\n",
            "Luger & Stubblefield (2004, pp. 591–632)\n",
            "\n",
            "Metadata:\n",
            "  id: 313\n",
            "  language: en\n",
            "  source: https://en.wikipedia.org/wiki/Artificial_intelligence\n",
            "  title: Artificial intelligence - Wikipedia\n",
            "--------------------------------------------------\n",
            "Document 5:\n",
            "Content:\n",
            "Retrieved from \"https://en.wikipedia.org/w/index.php?title=Artificial_intelligence&oldid=1238315903\"\n",
            "\n",
            "Metadata:\n",
            "  id: 531\n",
            "  language: en\n",
            "  source: https://en.wikipedia.org/wiki/Artificial_intelligence\n",
            "  title: Artificial intelligence - Wikipedia\n",
            "--------------------------------------------------\n"
          ]
        }
      ]
    }
  ]
}